The traditional methods for prediction of oil reservoir utilize the seismic attributes generated when seismic waves passing through the stratum and geologic drilling data to make a comprehensive judgment in combination with the traditional geophysical methods. However, this type of prediction methods has high cost of research and judgement and its accuracy strongly depends on the prior knowledge of the experts. To address the above issues, based on the seismic data of the Subei Basin of Jiangsu Oilfield, and considering the sparseness and randomness of oil-labeling samples, a multi-granularity temporal structure representation based outlier detection algorithm was proposed to perform the prediction by using the post-stack seismic trace data. Firstly, the multi-granularity temporal structures for the single seismic trace data was extracted, and the independent feature representations were formed. Secondly, based on extracting multiple granularity temporal structure representations, feature fusion was carried out to form the fusion representation of seismic trace data. Finally, a cost-sensitive method was utilized for the joint training and judgement to the fused features, so as to obtain the results of oil reservoir prediction for these seismic data. Experiments and simulations of the proposed algorithm were performed on an actual seismic data of Jiangsu Oilfield. Experimental results show that the proposed algorithm is improved by 10% on Area Under Curve (AUC) compared to both of the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) algorithms.